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ABSTRACT 


A  distinction  is  drawn  between  computer  based  instruction  of 
knowledge  systems  and  computer  based  instruction  of  dynamic  skills. 
There  is  reason  to  expect  that  the  findings  of  research  on  knowledge 
system  instruction  will  not  apply  universally  to  dynamic  skill 
instruction.  In  particular,  a  theory  of  cognitive  resource  demand 
suggests  that  the  principle  of  immediate  instructional  feedback  may  not 
apply  in  dynamic  skill  training.  Because  students  in  dynamic  skill 
training  are  often  heavily  loaded  with  processing  demands, 
instructional  feedback  must  be  postponed  until  the  students  have 
sufficient  free  resources  to  process  it.  This  hypothesis  was  tested  in 
an  experiment  in  computer  based  instruction.  One  group  of  students 
received  instructional  feedback  upon  request,  while  a  second  group 
received  feedback  under  program  control.  The  group  with  control  over 
feedback  made  significantly  fewer  errors  in  training  than  did  the  group 
that  did  not  control  timing  of  the  instructional  feedback  messages. 
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INTRODUCTION 


There  has  been  little  academic  research  on  computer  based  dynmaic 
skill  training  (see  Munro,  Towne,  &  Fehling,  1981).  Instead,  the  bulk 
of  academic  research  on  computer  based  learning  systems  has  dealt  with 
knowledge  system  teaching.  By  knowledge  systems  we  mean  coherent 
bodies  of  essentially  propositional  knowledge.  By  dynamic  skills  we 
mean  bodies  of  knowledge  that  coordinate  perception,  motor  skills,  and 
decision  making  in  real  time  driven  environments.  Existing  computer 
based  training  devices  for  dynamic  skills  are  typically  vehicle 
simulation  practice  environments.  Examples  include  aircraft  and 
helicopter  simulators  for  pilot  training.  Such  training  installations 
are  typically  very  expensive  (more  than  one  million  dollars  per  unit). 
They  are  typically  fully  utilized  for  training,  with  no  available  time 
for  research.  In  addition,  the  demands  of  verisimilitude  frequently 
result  in  simulator  designs  that  cannot  provide  the  flexibilty  of 
programming  required  by  computer  based  learning  research. 

Recent  advances  in  the  development  of  low  cost  microcomputer 
systems  make  possible  research  on  computer  based  dynamic  skill  training 
using  systems  developed  solely  for  that  research  purpose.  Such  a  low 
cost  computer  based  learning  research  tool  has  been  developed  in  our 
laboratory  (Munro,  Towne,  &  Fehling,  1981).  This  experimental  system 
for  the  study  of  dynamic  skill  training  is  being  used  to  conduct  a 
series  of  experiments  on  a  number  of  issues.  This  technical  report 
describes  the  first  experiment  in  a  planned  series. 

It  may  be  dangerous  to  assume  that  the  research  findings  on 
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computer  based  learning  of  traditional  knowledge  systems  will  apply  to 
dynamic  skill  training.  The  present  experiment  was  designed  to  explore 
certain  contrasts  between  dynamic  skill  training  and  knowledge  system 
teaching  in  computer  based  learning.  Two  principles  of  knowledge 
system  training  are  challenged  in  the  dynamic  skill  training 
environment.  The  first  of  these  is  that  instructional  feedback 
(knowledge  of  results)  should  be  immediate.  The  second  is  that  student 
control  over  instructional  pacing  or  sequence  does  not  ordinarily 
result  in  a  large  improvement  in  learning. 

The  Dynamic  Skill  Training  Task 

The  training  task  is  loosely  based  upon  the  task  of  Navy  Air 
Intercept  Controllers.  In  the  laboratory  task,  there  are  two  display 
screens,  one  representing  a  radar  screen,  the  other  the  display  console 
of  a  tracking  computer.  A  keyboard  with  fifteen  specially  labeled 
command  keys  and  a  numeric  keypad  is  used  for  student  input  to  the 
simulated  tracking  computer.  The  students  use  a  joystick  and  certain 
keyboard  keys  to  place  identifying  symbols  on  the  simulated  radar 
screen.  The  task  calls  for  close  attention  to  the  simulated  radar 
screen,  which  is  updated  once  every  eight  seconds.  Students  must 
quickly  execute  a  series  of  key  presses  in  response  to  certain  events 
observed  on  the  radar  screen,  such  as  the  appearance  of  a  new  blip  or 
the  movement  of  simulated  friendly  and  enemy  aircraft  into  a  particular 
proximity.  In  addition,  the  student  must  keep  track  of  certain  tasks 
that  are  periodically  required,  such  as  determining  fuel  and  weapons 
status  for  the  simulated  aircraft. 

There  are  many  opportunities  for  errors  in  this  task.  When  the 
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training  system  detects  a  student  error,  it  can  present  information 
about  the  error  in  a  reserved  area  of  the  simulated  tracking  computer 
console  screen.  In  the  first  experiment,  two  different  methods  of 
determining  when  to  present  these  messages  were  employed.  For  the 
intrusive  feedback  group,  these  error  messages  were  automatically 
presented  at  the  time  that  the  system  detected  the  errors.  For  the 
non-intrusive  feedback  group,  an  error  message  was  shown  only  when 
requested. 

A  student’s  interaction  with  the  Air  Intercept  Controller  training 
system  has  parallels  to  a  conversational  interaction  between  two 
people.  In  normal  conversation,  the  participants  exchange  indications 
of  their  readiness  to  accept  input  from  each  other:  they  signal 
turn-taking  in  the  conversation.  In  conventional  knowledge  system 
teaching  in  CBI,  each  student  response  (such  as  typing  in  the  answer  to 
a  question)  can  be  thought  of  as  a  signal  that  the  student  has 
completed  and  surrendered  a  conversational  turn.  After  making  a 
response,  the  student  expects  a  reply  from  the  teaching  system.  In 
dynamic  skill  training,  however,  each  student  interaction  with  the 
simulation  system  is  not  a  signal  that  the  student  is  surrendering  a 
turn.  Instead,  the  student  reamains  actively  engaged  with  the 
simulation,  preparing  a  series  of  actions  in  response  to  observed  and 
expected  events.  In  this  context,  the  interjection  of  an 
instructional  feedback  message  is  an  intrusion.  The  system  is,  in  some 
sense,  breaking  in  on  the  student's  turn. 

This  "conversational  rules”  hypothesis  is  just  a  special  case  of 
an  attentional  demand  hypothesis.  In  a  complex  training  session,  most 
of  a  student's  cognitive  processing  resources  are  likely  to  be 
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allocated  to  attending  to  and  responding  to  the  task  Itself.  If  the 
task  Is  suddenly  intruded  upon  by  an  instructional  message,  the 
intrusion  will  demand  additional  processing  resources  to  perform  the 
attentional  shift.  This  surge  in  processing  resource  demand  is  likely 
to  interfere  with  the  normal  learning  and  performance  processes.  If 
the  disruption  occurs  at  a  point  in  the  task  when  a  large  percentage  of 
cognitive  resources  are  already  committed,  then  either  attention  to 
feedback  or  performance  on  the  task  is  likely  to  suffer.  The 
attentional  demand  hypothesis  predicts  that  feedback  given  intrusively 
will  be  more  disruptive  of  learning  than  will  feedback  provided 
non-intrusively . 

Conversational  cooperation  can  be  viewed  as  an  adaptation  to 
information  processing  resource  limitations.  A  dynamic  skills  training 
system  that  partially  emulates  some  of  the  features  of  conversational 
turn-taking  in  instructional  feedback  should  prove  superior  to  a  system 
that  interrupts  student  task  processing  with  instructional  messages. 

In  the  experiment,  cooperative  turn-taking  is  emulated  by  signaling  a 
readiness  to  provide  feedback  in  a  non-intrusive  way,  and  then 
postponing  the  presentation  of  the  message  until  the  student  explicitly 
surrenders  a  turn  and  requests  presentation  of  feedback. 

Argument  can  be  found  to  predict  that  the  non-intrusive  feedback 
condition  (student  paced  presentation  of  feedback)  could  present 
difficulties  for  representational  (as  opposed  to  attentional) 
processes.  Students  in  the  non-intrusive  group  frequently  postpone 
viewing  a  feedback  message  until  some  time  after  the  commission  of  the 
error.  The  results  of  studies  on  delay  of  reinforcement  during 
learning  can  be  interpreted  to  imply  that  such  a  delay  will  be 
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detrimental  to  learning.  According  to  this  argument,  feedback  serves 
as  a  reinforcer,  and,  therefore,  the  delays  which  occur  in  the 
non-intruslve  condition  should  Impair  overall  student  performance.  A 
cognitive  view  of  this  line  of  reasoning  is  that  when  the  feedback 
message  is  delayed  it  is  more  likely  that  the  content  of  such  a  message 
will  not  be  correctly  related  to  the  student's  representation  of  the 
task.  The  student  may  not  understand  to  which  context  the  feedback 
message  refers.  On  the  other  hand,  a  student  whose  feedback  is 
presented  intrusively  always  receives  the  feedback  message  immediately 
following  the  context  for  which  it  is  generated  and  so  should  not 
suffer  this  representational  (or  reinforcement)  disadvantage. 

We  do  not  expect  the  potential  representational  disadvantage  for 
non-intrusive  group  students  to  be  as  important  as  the  attentional 
disadvantage  for  intrusive  group  students.  We  postulate  that  students 
have  an  ability  to  remember  the  context  to  which  a  feedback  message 
refers  from  the  content  of  that  message.  The  research  specifically 
addressed  to  delay  of  knowledge  of  results  can  be  interpreted  to 
support  this  point  of  view  (Kling  &  Schrier,  1971).  In  these  studies 
it  appeared  that  making  students  wait  for  feedback  was  not  detrimental 
if  little  or  no  responding  was  required  during  the  wait  interval.  This 
suggests  that  students  were  able  to  maintain  a  memory  of  the  context 
for  the  knowledge  of  results  information.  If  students  in  our  Air 
Intercept  Controller  training  task  were  similarly  able  to  recall  the 
context  of  a  feedback  message,  then  it  would  follow  that  students 
recieving  feedback  non-intrusively  would  not  suffer  a  representational 
or  reinforcement  disadvantage. 
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The  Experiment 

Method 

Subjects.  Subjects  were  paid  volunteers  who  responded  to  posted 
notices  and/or  class  announcements  made  at  two  colleges  (El  Camino 
College  and  California  State  University,  Dominguez  Hills)  and  one  high 
school  (Redondo  Beach  High  School).  Thirty-five  subjects  participated 
in  the  experiment.  Of  these,  thirty  completed  the  experimental 
training  task.  Three  subjects  were  dropped  from  the  experiment  due  to 
poor  performances.  Two  of  these  failed  to  correctly  follow  directions 
in  the  preliminary  training  phase,  before  the  practice  training  of 
interest.  One  was  dropped  because  of  failure  on  all  30  practice 
problems,  including  those  that  were  performed  successfully  by  all  other 
students.  Two  of  the  thirty-five  subjects  chose  to  discontinue  the 
experiment  before  completion.  Subjects  were  assigned  to  one  of  the  two 
experimental  groups  in  alternating  order  as  they  arrived  for  the 
experiment.  Each  subject  who  completed  the  training  received  fifteen 
dollars.  Students  who  chose  to  terminate  their  participation  or  who 
were  dropped  due  to  poor  performance  received  either  five  or  ten 
dollars,  depending  on  the  length  of  their  participation. 

Procedure.  Subjects  were  run  individually  in  the  experiment. 
Completion  of  the  training  session  required  from  three  hours  twenty 
minutes  to  four  and  one-half  hours.  All  subjects  first  viewed  a  six 
minute  videotaped  explanation  and  demonstration  of  the  Air  Intercept 
Controller  task.  They  then  were  instructed  in  the  functions  of  each  of 
the  control  devices  used  in  the  simulated  task— thirteen  specially 
labeled  keyboard  keys  and  a  joystick— by  a  computer-based-training 
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program  called  Pre-AIC.  The  Pre-AIC  program  consisted  of  a  series  of 
text  presentations  describing  the  task  in  greater  detail  than  had  been 
presented  in  the  videotaped  introduction.  It  also  presented  simulation 
segments  that  the  student  was  required  to  interact  with  by  using  the 
control  keys  and  joystick.  Students  who  failed  to  complete  two  or  more 
of  the  twelve  training  modules  were  dropped  from  the  experiment.  Two 
of  the  thirty-five  subjects  were  disqualified  from  the  experiment  by 
this  criterion. 

After  completing  the  Pre-AIC  computer-based  instruction  program, 
all  students  then  viewed  the  same  videotaped  sequence  reviewing  the  Air 
Intercept  Controller  task  and  stating  the  requirements  of  the  task. 

This  tape  segment  lasted  eight  minutes.  At  this  point  the  treatment  of 
students  in  the  two  groups  diverged.  Each  group  viewed  a  videotape 
segment  describing  the  way  in  which  instructional  feedback  would  be 
presented  and  how  they  should  respond.  This  tape  segment  lasted  one 
minute  and  twenty  seconds  for  the  intrusive  feedback  group  and  three 
minutes  and  thirty  seconds  for  the  non-intrusive  feedback  group. 
Students  were  then  given  a  five  minute  break  from  the  training  task. 

At  the  end  of  the  break,  students  were  given  practice  in  the  Air 
Intercept  Controller  task,  using  a  simulator  trainer  program  called 
AIC.  The  AIC  program  presented  a  series  of  30  problems  to  the  student, 
organized  in  four  banks  of  five,  ten,  ten,  and  five  problems. 

Difficulty  was  held  roughly  constant  within  each  bank,  but  increased 
with  the  progression  of  problem  banks.  Students  in  both  groups 


received  the  same  problems,  and  the 
students  in  the  two  groups  in  every 


training  program  was  the  same  for 
respect  except  instructional 


) 


1  / 


feedback 


Page 


Instructional  Feedback  Treatments.  The  AIC  program  continually 
monitored  student  performance  for  a  variety  of  errors.  Examples 
include  inacurrately  positioning  a  symbol  on  the  simulated  radar 
screen,  or  failing  to  get  a  fuel  status  update  from  the  pilots  of  the 
simulated  aircraft  within  the  required  time.  The  Appendix  contains  a 
complete  list  of  these  errors.  For  both  feedback  conditions,  when  the 
AIC  program  detected  an  error,  a  warning  tone  sounded  and  the  word 
"Advisory"  appeared  in  an  area  of  the  computer  console  display  reserved 
for  instructional  messages. 

At  this  point,  those  students  in  the  intrusive  feedback  group 
were  presented  with  a  one-  to  four-line  instructional  message  related 
to  the  error  Just  detected.  While  the  message  was  displayed,  the 
simulation  was  frozen.  The  radar  screen  did  not  change,  and  all  the 
normally  active  keys  of  the  computer  console  were  dead.  Only  one  key, 
the  "Accept  message"  key  could  be  pressed  by  the  student  with  effect. 
When  this  key  was  pressed,  the  instructional  feedback  message  was 
erased  from  the  screen  and  the  problem  continued,  resuming  at  the  point 
at  which  it  had  been  frozen.  If  more  than  one  error  had  been  detected 
at  one  time,  they  were  presented  in  sequence,  from  most  to  least 
recent,  as  the  student  pressed  the  "Accept  message"  key,  until  all  were 
seen  by  the  student.  After  the  last  currently  active  message  was  seen, 
the  word  "Advisory"  was  erased  from  the  screen  along  with  the  last 
message. 

The  students  in  the  non- intrusive  feedback  group  were  not 
immediately  presented  with  the  instructional  message  after  the  system 
sounded  the  error  tone  and  displayed  "Advisory"  in  the  reserved  area. 
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Thus,  unlike  the  students  in  the  intrusive  condition,  the  non-intrusive 
Feedback  Group  members  were  able  to  choose  the  time  of  the  appearance 
of  the  error  messages  by  depressing  a  special  "Help"  key.  Depressing 
this  key  caused  the  error  message  to  appear  and  the  simulation  to 
freeze  until  the  student  pressed  the  "Accept  message"  key.  If  more 
than  one  error  had  been  detected  by  the  system  before  the  feedback 
message  was  requested,  then  the  student  was  first  presented  with  the 
most  recent  error  message.  In  each  case,  depressing  the  "Accept 
feedback"  key  caused  the  error  message  to  be  erased  and  the  simulation 
to  resume.  When  all  pending  feedback  messages  had  been  presented,  the 
word  "Advisory"  was  removed.  If,  at  the  end  of  a  problem,  the  student 
had  not  reviewed  messages  for  all  the  errors  detected  by  the  system, 
then  the  student  was  given  the  option  of  seeing  these  messages  before 
beginning  the  next  problem. 

In  summary,  students  in  the  intrusive  feedback  group  were 
presented  with  an  error  message  for  each  detected  new  error  at  the  time 
that  the  AIC  program  recognized  the  error.  Students  in  the 
non-intrusive  feedback  group  had  the  option  of  determining  when  and 
whether  they  would  view  the  error  messages. 

Data  collection.  The  AIC  simulation  training  program  preserved 
an  exhaustive  record  of  each  student's  interactions  with  the  program. 
These  data  sets  were  later  processed  by  data  extraction  programs  to 
produce  records  of  errors,  time  on  problems,  and  other  variables  of 
interest. 
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Results 

Errors.  Number  of  errors  per  problem  was  used  as  one  measure  of 
learning.  Table  1  presents  an  analysis  of  variance  of  the  error  data. 
The  mean  number  of  errors  for  the  students  In  the  non-intrusive  group 
was  9.17*  and  for  the  Intrusive  treatment  group,  the  mean  number  of 
errors  per  problem  was  15.67.  This  difference  was  highly  significant, 
suggesting  that  students  in  the  non-intrusive  group  learned  more  than 
those  in  the  intrusive  group. 

Errors,  last  ten  problems.  A  second  analysis  was  performed,  using 
only  the  error  data  from  the  last  one-third  of  the  thirty  practice 
problems.  These  problems,  which  were  longer  and  more  difficult  than 
most  of  the  previous  problems,  were  treated  as  a  measure  of  final 
training  performance  in  this  analysis.  Students  in  the  non-intrusive 
feedback  group  made  a  mean  of  1M.74  errors  per  problem  in  these  ten 
problems.  In  the  intrusive  group,  the  mean  was  22.82  errors  per 
problem. 

It  is  possible  that  the  comparison  of  total  errors  made  in 
practice  training,  given  in  Table  1,  is  not  the  best  measure  of 
learning.  If  students  learn  from  making  errors,  then  fewer  errors  in 
training  might  not  portend  better  performance  on  the  final  task.  After 
twenty  problems,  however,  some  training  has  taken  place.  Yet  the  data 
of  Table  2  reveal  that  the  students  in  the  non-intrusive  group  made 
fewer  errors  in  the  last  ten  problems  than  those  in  the  intrusive 
group.  This  result  suggests  that  non-intrusive  feedback  promotes 
learning  in  dynamic  skill  training. 
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Tine  per  or obi an.  The  tine  spent  on  each  problen  by  each  student 
was  recorded.  Table  3  presents  the  analysis  of  the  tine  on  problen 
data,  where  tine  is  expressed  in  tenths  of  seconds.  Intrusive  group 
students  spent  a  nean  of  264.7  seconds  per  problen,  and  non-intrusive 
group  students  took  274.3  seconds.  This  difference  was  not 
significant . 

It  had  been  predicted  that  students  in  the  non-intrusive  group 
would  require  significantly  less  time  per  problem  than  those  in  the 
intrusive  group.  In  retrospect,  it  can  be  seen  that  this  prediction  is 
in  partial  conflict  with  the  prediction  of  fewer  errors  for  the  non- 
intrusive  group.  Some  errors,  such  as  allowing  one's  fighters  to  be 
shot  down  by  the  enemy,  resulted  in  premature  termination  of  problems. 
Because  intrusive  group  students  are  more  likely  to  make  such  errors, 
a  prediction  of  shorter  time  per  problem  for  intrusive  group  students 
would  have  been  appropriate.  In  any  case,  the  experimental  results  do 
not  permit  the  rejection  of  the  null  hypothesis  that  there  is  no 
difference  in  time  to  complete  problems  for  the  two  groups. 

Crucial  and  non-crucial  errors.  Student  errors  are  classified  by 
the  AIC  program  into  thirty-six  types.  Of  these,  twenty-three  may  be 
termed  "crucial"  errors,  in  that  they  are  likely  to  materially  affect 
the  student's  chances  of  "winning"  an  exercise  by  shooting  down  the 
enemy  aircraft.  The  other  thirteen  types  of  errors  are  non-crucial  in 
that  they  reflect  errors  of  form  that  will  not  immediately  decrease  the 
chances  of  winning  the  problem.  Table  4  presents  the  analysis  of 
crucial  errors  for  all  thirty  problems.  It  shows  a  small  but 
significant  difference  in  number  of  crucial  errors  per  problem  between 
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the  two  groups.  The  non-intrusive  feedback  group  made  an  average  5.36 
crucial  errors  per  problem,  while  the  intrusive  group  made  6.14.  Table 
5  shows  a  significant  and  much  larger  difference  in  non-cruclal  errors. 
The  mean  number  of  non-cruclal  errors  per  problem  for  the  non-intrusive 
training  group  was  3.18.  For  the  intrusive  feedback  group,  the  mean 
was  9.53  errors  per  problem.  These  results  suggest  that,  even  though 
overloaded  by  the  intrusive  instructional  messages,  the  intrusive  group 
students  are  still  able  to  Jeeide  which  performance  factors  to  attend 
to.  They  choose  to  permit  greater  deterioration  of  their  non-crucial 
performance  rather  than  t'^ir  crucial  performance. 

Joystick  errors.  Vn  an  attempt  to  determine  what  kind  of 
performance  is  affected  by  the  intrusiveness  of  instructional  feedback, 
a  separate  analysis  of  joystick  errors  was  performed.  Most  of  the  AIC 
task  requires  the  fusion  of  skills  of  planning,  time  or  distance 
estimation,  and  decision  making,  as  well  as  some  motor  coordination. 

One  essential  skill,  however,  is  primarily  one  of  motor  coordination  . 
This  is  the  task  of  using  the  Joystick  and  keyboard  to  "hook"  a  symbol 
on  a  simulated  radar  screen  blip.  Table  6  shows  that  there  was  no 
significant  difference  in  the  total  number  of  joystick  errors  made  by 
students  in  the  two  instructional  treatment  groups  over  the  course  of 
the  thirty  practice  problems.  This  implies  that  the  deleterious  effect 
of  intrusive  feedback  may  not  equally  degrade  all  types  of  skills.  The 
motor  skill  of  using  the  joystick  appears  not  to  be  harmed  by  the 
processing  loads  imposed  by  intrusive  feedback. 
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Discussion 

The  results  support  our  hypothesis  that  the  processing  demands  of 
dynamic  skill  simulation  training  require  non-intrusive  rather  than 
intrusive  feedback.  Students  trained  with  the  intrusive  feedback 
method  committed  significantely  more  errors,  both  in  the  last  set  of 
problems  and  throughout  the  practice  training,  than  did  those  trained 
by  the  non-intrusive  method.  Intrusive  group  students  made 
significantly  more  errors  of  both  crucial  and  non-crucial  types.  This 
higher  incidence  of  errors  is  evidence  that,  in  this  dynamic  skill 
training  system,  students  who  receive  intrusive  instructional  feedback 
learn  less  well  than  those  who  have  control  over  when  the  feedback  will 
be  presented. 

A  conversational  analogy  to  computer  based  training  is  presented 
above.  In  normal  conversation,  the  participants  take  turns  talking, 
exchanging  special  signals  to  indicate  a  willingness  to  surrender  a 
turn  or  to  request  a  turn  as  soon  as  possible.  The  non-intrusive 
feedback  condition  simulates  this  aspect  of  conversation.  The  training 
system  signals  that  it  has  a  feedback  message  by  sounding  a  tone.  The 
student  surrenders  a  turn  by  depressing  a  special  key.  These  features 
allow  the  student  to  control  the  system's  instructional  input. 

The  fact  that  our  results  indicate  a  superiority  for  a  treatment 
condition  that  gives  the  students  control  over  instructional 
presentation  contrasts  markedly  with  the  results  of  studies  of  student 
oontrol  of  conventional  knowledge  system  instruction  in  CBI.  For 
example,  studies  by  Judd,  O'Neil,  &  Spelt  (1974);  Lahey,  Hurlock,  & 
McGrain  (1973);  and  White  &  Smith  (1974)  all  find  no  performance 
improvement  due  to  student  oontrol  of  instruction  in  computer  based 
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instruction.  It  is  not  surprising  that  the  effect  is  not  found  in 
these  cases,  but  is  found  in  dynamic  skill  simulation  training.  In 
conventional  computer  based  instruction,  students  consciously  surrender 
a  turn  in  the  "conversational"  interaction  every  time  they  depress  the 
Return  key. 

The  conversational  analogy  is  just  one  way  of  viewing  the 
difference  between  student  control  or  lack  of  control  of  attentional 
demands.  If  the  student  can  control  the  cognitive  processing  load  by 
postponing  instructional  feedback  until  the  processing  required  by  the 
task  is  at  a  low  level,  then  more  resources  will  be  available  to 
process  the  instructional  message.  Furthermore,  the  resources  devoted 
to  the  dynaic  task  during  a  demanding  portion  of  the  episode  will  not 
be  diverted  to  processing  feedback  messages.  Further  research  is 
called  for  to  determine  which  of  these  effects  is  primarily  responsible 
for  improved  learning  in  a  non-intrusive  feedback  environment.  It  is 
possible  that  improved  attention  to  instruction  and  to  the  task  both 
have  beneficial  consequences  for  learning  dynamic  skills. 

One  type  of  error,  joystick  manipulation  errors,  required  only 
perceptual  and  motor  skills,  rather  than  planning  or  decision  making 
skills.  Students  in  the  two  treatment  groups  did  not  differ 
significantly  in  the  number  of  this  type  of  error.  This  result  implies 
that  this  perceptual-motor  skill  complex  was  less  sensitive  to  the 
processing  load  imposed  by  the  dynaic  task  together  with  instructional 
feedback  processing  than  were  the  decison-making  and  planning  skills 
called  for  by  the  task.  Further  research  is  called  for  to  determine 
the  detailed  consequences  of  processing  load  on  performance  in  dynamic 
skill  training  tasks  such  as  the  simulated  Air  Intercept  Controller 
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Task  used  in  this  experiment. 

More  research  is  also  called  for  to  determine  other  means  of 
improving  dynamic  skill  Instruction  by  reducing  or  by  more  effectively 
distributing  processing  loads  during  training.  Experiments  have  not 
yet  been  performed  to  explore  the  consequences  of  manipulating  the 
continuity  of  the  simulation  in  training.  (See  Munro,  et  al.f  1981, 
for  a  plan  of  research  in  this  area.)  In  addition,  the  use  of 
alternate  presentation  modes  for  instruction  and  simulation  should  be 
explored.  Research  on  selective  attention  has  shown  that  simultaneous 
attention  is  facilitated  by  distinguishing  the  channels  of  information 
as  much  as  possible.  This  suggests  the  need  to  experiment  with 
different  modalities  for  task  performance  and  for  feedback.  In  the 
case  of  a  system  such  as  ours,  the  dynamic  task  is  largely  visual 
(textual)  and  tactile  (key  presses  and  joystick  manipulation).  If 
feedback  presentations  were  by  means  of  computer-generated  voice 
output,  rather  than  text,  then  the  discriminability  between  the  task 
and  Instruction  would  be  improved.  It  is  possible  that  such  enhanced 
discriminability  would  reduce  processing  demands  and  thereby  improve 
learning. 
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Summary 

The  resuls  of  the  intrusive  feedback  experiment  with  the  AIC 
simulation  training  system  do  not  accord  with  two  commonly  accepted 
precepts  of  computer  based  learning.  The  first  of  these  is  that 
Instructional  feedback  should  be  as  prompt  as  possible.  In  this 
experiment,  the  group  that  received  immediate  feedback  performed 
significantly  less  well  than  the  group  that  had  the  delayed  feedback 
option.  The  second  expectation  was  that  student  control  over  the 
instructional  process  would  have  little,  if  any,  effect.  In  this 
experiment,  the  group  that  chose  the  timing  of  instructional  feedback 
presentations  made  significantly  fewer  errors  than  the  group  that  did 
not  have  this  control.  This  finding  is  not  anomalous  in  view  of  the 
differences  in  processing  demand  between  conventional  computer  based 
instruction  and  dynamic  skill  training.  The  transient  processing 
demand  fluctuations  of  dynamic  skill  training  can  be  exploited  by 
giving  students  control  over  the  timing  of  some  aspect  of  instruction. 

The  most  important  conclusion  to  be  drawn  from  this  study  is  that 
the  findings  of  computer  based  learning  research  on  knowledge  system 
training  may  not  be  applicable  to  dynamic  skill  training.  Further 
research  is  called  for  to  determine  the  characteristics  of  an 
appropriate  training  methodology  for  computer  based  simulation 
training. 
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Intrusive  Feedback 

Group 

Non-intrusive  Feedback 

Group 

15.67 

9.17 

2-WAY  ANOVA 


Source  of  Variation 

Sum  of  Squares 

DF 

Mean  Square 

F 

Group 

9423.297 

1 

9423.297 

74.721 

Problem 

37947.422 

29 

1308.532 

10.376 

Group  x  Problem 

3618.318 

29 

124.770 

0.989 

Residual 

100511.750 

797 

126.113 

Table  1 

Total  Errors.  Analysis  of  Variance 
*  P  < .01 


**  P  <.001 


age  19 


Intrusive  Feedback 
Group 

Non-intrusive  Feedback 

Group 

22.82 

14.74 

2-WAY  ANOVA  (Last  10  Problems) 


Source  of  Variation 

Sum  of  Squares 

DF 

Mean  Square 

F 

Group 

4941.145 

1 

4941.145 

75.168 

Problem 

14059.820 

9 

1562.202 

23.765 

Group  x  Problem 

736.404 

9 

81.823 

1.245 

n.s. 

Residual 

17091.066 

260 

65.735 

Table  2 

Errors,  Last  10  Problems 
Analysis  of  Variance 
**  P  <  .001 
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Intrusive  Feeeback 

Group 

Non-intrusi ve  Feedback 

Group 

2647 

2743 

2-WAY  ANOVA 


Source  of  Variation 

Sun  of  Squares 

DF 

Mean  Square  F 

Group 

1531126.000 

1 

1531126.000  3.117  n.s. 

Problem 

430203136.000 

29 

14834590.000  30.201  * 

Group  x  Problem 

13610033.000 

29 

469311.438  0.955  n.s. 

Residual 

391969024. C00 

798 

491189.250 

Table  3 

Time  per  Problem.  Analysis  of  Variance. 
*  P<  .01 
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Intrusive  Feedback 

Group 

Non-intrusive  Feeeback 

Group 

6.14 

5.36 

2-WAY  ANOVA  FOR  CRUCIAL  ERRORS 


Source  of  Variation 

Sum  of  Squares 

DF 

Mean  Square 

F 

Group 

151.928 

1 

151.928 

17.947  ** 

Problem 

12426.016 

29 

428.483 

50.616  * 

Group  x  Problem 

197.630 

29 

6.815 

0.805  n.s. 

Residual 

6746.871 

797 

8.465 

Table  4 

Crucial  Errors.  Analysis  of  Variance. 
*  P  <.01 
**  P  <.001 
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Intrusive  Feedback 
Group 

Non-intrusive  Feedback 
Group 

9.53 

3.81 

A  2-WAY  ANOVA  FOR  NON-CRUCIAL  ERRORS 


Source  of  Variation 

Sum  of  Squares 

DF 

Mean  Square 

F 

Group 

7120.387 

1 

7120.387 

67.743  ** 

Problem 

9352.324 

29 

322.494 

3.068  ** 

Group  x  Problem 

2967.144 

29 

102.315 

0.973  n.s. 

Residual 

83772.313 

797 

105.110 

Table  5 

Non-Crucial  Errors.  Analysis  of  Variance 
**  P  <.001 
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Mean  Joystick  Errors  per  Student 


Intrusive  Feedback 
Group 

Non- intrusive  Feedback 
Group 

31.0 

26.6 

T-TEST  FOR  JOYSTICK  ERRORS 


T 

(pooled) 

1.03 

DF 

28 

T 

(not  pooled) 

1.03 

DF 

29.96 

F 

ratio= 

0.93 

DF 

14, 

Not  Significant 


Table  6 

Joystick  Errors.  T-Test  . 
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APPENDIX 

Types  of  Student  Errors  Detected 

.gru.Sj.at  Errors 

1  Mishook.  To  depress  a  reserved  Symbol  key  when  the  joystick 
cursor  is  not  centered  (within  the  error  margin)  on  a  blip. 

2  Fail  to  hook  CAP  after  appear.  To  fail  to  label  a  CAP  within 
60  seconds  of  its  first  appearance. 

3  Fail  to  rehook  CAP  after  first  hook.  To  fail  to  relabel  a  CAP 
within  30  seconds  of  first  labeling  it.  The  blip  labeled  by  the 
CAP  symbol  must  have  moved  since  the  first  hook. 

k  Fail  to  hook  Bogey  after  appear.  To  fail  to  label  a  Bogey 
within  2k  seconds  of  its  first  appearance. 

5  Fail  to  rehook  Bogey  after  first  hook.  To  fail  to  relabel  a 
Bogey  within  18  seconds  of  first  labeling  it.  The  blip  labeled 
by  the  Bogey  symbol  must  have  moved  since  the  first  hook. 

6  Fail  to  rehook  Bogey  after  a  heading  jink.  To  fail  to  rehook  a 
Bogey  blip  within  36  seconds  after  it  makes  a  change  of  course. 

7  Fail  to  rehook  Bogey  after  a  speed  Jink.  To  fail  to  rehook  a 
Bogey  blip  within  36  seconds  of  its  change  of  speed. 

8  Incorrect  pairing  for  intercept  heading.  To  choose  to  assign  a 
CAP  to  an  intercept  when  there  is  a  closer,  unassigned  CAP  to 
the  target  Bogey. 

9  Fail  to  elicit  intercept  heading.  To  fail  to  elicit  an 
intercept  heading  to  a  Bogey  within  18  seconds  of  rehooking  the 
Bogey  blip,  when  there  is  a  free  CAP  available  for  the 
assignment. 

10  Incorrect  pairing  for  attack  heading.  To  choose  to  assign  a 
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CAP  to  attack  a  Bogey  when  there  is  a  closer,  unassigned  CAP  to 
the  target  Bogey. 

11  Fail  to  elicit  an  attack  heading.  To  fail  to  elicit  an  attack 
heading  to  a  Bogey  within  18  seconds  of  a  CAP  entering  attack 
range,  if  the  CAP  is  free  or  assigned  to  the  Bogey. 

12  Fail  to  elicit  an  attack  heading.  To  fail  to  elicit  an  attack 
heading  to  a  Bogey  within  18  seconds  of  rehooking  the  Bogey 
while  it  is  in  attack  range  of  the  CAP  assigned  to  it. 

13  Send  wrong  intercept  heading.  After  eliciting  an  intercept 
heading,  to  send  a  different  intercept  than  that  recommended. 

1M  Fail  to  send  intercept  heading.  To  fail  to  send  an  intercept 
heading  for  more  than  12  seconds  after  eliciting  the  intercept 
heading.  Applies  only  if  both  CAP  and  Bogey  are  still  alive  at 
time  of  error. 

15  Fail  to  reassign  CAP  after  splashing  Bogey.  To  fail  to  either 
elicit  a  new  intercept  or  attack  heading  or  to  abort  the  CAP 
flight  for  more  than  18  seconds  after  shooting  down  its  Bogey. 

16  Send  wrong  attack  heading.  After  eliciting  an  attack  heading, 
to  send  a  different  attack  heading  than  recommended. 

17  Fail  to  send  attack  heading.  To  fail  to  send  the  attack  heading 
for  more  than  12  seconds  after  eliciting  it.  Applies  only  if 
both  Bogey  and  CAP  are  still  up. 

18  Get  shot  down.  Fail  to  fire  when  in  firing  range.  When  range 
is  very  close,  Bogey  shoots  down  CAP. 

19  Fire  when  not  on  attack  heading. 

20  Fail  to  fire  in  firing  range.  More  than  12  seconds  pass 
without  firing  since  CAP  enters  firing  range. 
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Crucial  Errors 

1  Fail  to  rehook  CAP  after  turn  to  intercept.  To  fail  to  rehook  CAP 
for  more  than  36  seconds  after  it  turns  to  a  new  intercept 
heading. 

2  Fail  to  rehook  CAP  after  turn  to  attack.  To  fail  to  rehook  CAP 
for  more  than  36  seconds  after  it  turns  to  a  new  attack 
heading. 

3  Send  attack  heading  when  CAP  not  in  attack  range  of  Bogey. 

4  Fire  when  CAP  out  of  firing  range. 

5  Fire  when  CAP  out  of  weapons. 

6  Abort  CAP  flight  prematurely.  To  send  CAP  back  toward  home  base 
while  there  is  still  a  Bogey  on  the  screen  and  the  CAP  has 
enough  fuel. 

7  Fail  to  get  first  fuel  update.  To  fail  to  elicit  information  on 
fuel  supply  within  60  seconds  of  first  rehook  of  CAP. 

8  Fail  to  get  subsequent  fuel  update.  To  fail  to  elicit  fuel  supply 
information  within  60  seconds  of  previous  fuel  update. 

9  Fail  to  get  first  weapons  update.  To  fail  to  elicit  information 
on  weapons  quantity  within  60  seconds  of  first  rehook  of  CAP. 

10  Fall  to  get  subsequent  weapons  update.  To  fail  to  elicit  weapons 
information  within  60  seconds  of  previous  weapons  update. 

11  Send  intercept  heading  before  eliciting  intercept  heading  for 
CAP. 

12  Send  attack  heading  before  eliciting  attack  heading  for  CAP. 


SC/MUNRO  October  15,  1981 


Page  1 


Navy 


1  Dr.  Ed  Aiken 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Arthur  Bachrach 

Environmental  Stress  Program  Center 
Naval  Medical  Research  Institute 
Bethesda,  MD  20014 

1  Meryl  S.  Baker 
NPRDC 
Code  P309 

San  Diego,  CA  92152 

1  CDR  Thomas  Berghage 

Naval  Health  Research  Center 
San  Diego,  CA  92152 

1  Dr.  Alvah  Bittner 

Naval  Biodynamics  Laboratory 
New  Orleans,  Lousiana  70189 

1  Dr.  Robert  Breaux 
Code  N-711 
NAVTRAEQUIPCEN 
Orlando,  FL  32813 

1  Chief  of  Naval  Education  and  Training 
Liason  Office 

Air  Force  Human  Resource  Laboratory 
Flying  Training  Division 
WILLIAMS  AFB,  A2  85224 

1  CDR  Mike  Curran 

Office  of  Naval  Research 
800  N.  Quincy  St. 

»  Code  270 

Arlington,  VA  22217 

1  DR.  PAT  FEDERICO 

NAVY  PERSONNEL  RAD  CENTER 
SAN  DIEGO.  CA  92152 

t 

1  Dr.  John  Ford 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 


Navy 


1  LT  Steven  D.  Harris,  MSC,  USN 
Code  6021 

Naval  Air  Development  Center 
Warminster,  Pennsylvania  18974 

1  Dr.  Lloyd  Hitchcock 

Human  Factors  Engineering 
Division  (6022) 

Naval  Air  Development  Center 
Warminster,  PA  18974 

1  Dr.  Jim  Hollan 
Code  304 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  CDR  Charles  W.  Hutchins 

Naval  Air  Systems  Command  Hq 
AIR-340F 
Navy  Department 
Washington,  DC  203 61 

1  Dr.  Norman  J.  Kerr 

Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 

1  Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command ,  Code  00A 
Pensacola,  FL  32508 

1  CAPT  Richard  L.  Martin,  USN 
Prospective  Commanding  Officer 
USS  Carl  Vinson  (CVN-70) 

Newport  News  Shipbuilding  and  Drydock  Co 
Newport  News,  VA  23607 

1  Dr.  George  Moeller 

Head,  Human  Factors  Dept. 

Naval  Submarine  Medical  Research  Lab 
Groton,  CN  06340 

1  Dr  William  Montague 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 
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Navy 


1  Commanding  Officer 

U.S.  Naval  Amphibious  School 
Coronado,  CA  92155 

1  Ted  M.  1.  Yellen 

Technical  Information  Office,  Code  201 
NAVY  PERSONNEL  R&D  CENTER 
SAN  DIEGO,  CA  92152 

1  Library,  Code  P201L 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

1  Technical  Director 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

6  Command ing  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Psychologist 

ONR  Branch  Office 
Bldg  114,  Section  D 
666  Summer  Street 
Boston,  MA  02210 

1  Psychologist 

ONR  Branch  Office 
536  S.  Clark  Street 
Chicago,  IL  60605 

1  Office  of  Naval  Research 
Code  437 

800  N.  Quincy  SStreet 
Arlington,  VA  22217 

1  Office  of  Naval  Research 
Code  441 

800  N.  Quincy  Street 
Arlington,  VA  22217 

5  Personnel  &  Training  Research  Programs 
(Code  458) 

Office  of  Naval  Research 
Arlington,  VA  22217 


1  Psychologist 

ONR  Branch  Office 
1030  East  Green  Street 
Pasadena,  CA  91101 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  &  Studies  Branch 
(OP-115) 

Washington,  DC  20350 

1  LT  Frank  C.  Petho,  MSC,  USN  (Ph.D) 

Selection  and  Training  Research  Division 
Human  Performance  Sciences  Dept. 

Naval  Aerospace  Medical  Research  Laborat 
Pensacola,  FL  32508 

1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Roger  W.  Remington,  Ph.D 
Code  152 
NAMRL 

Pensacola,  FL  32508 

1  Dr.  Worth  Scanland,  Director 

Research,  Development,  Test  &  Evaluation 
N-5 

Naval  Education  and  Training  Command 
NAS,  Pensacola,  FL  32508 

1  Dr.  Sam  Schiflett,  SY  721 

Systems  Engineering  Test  Directorate 
U.S.  Naval  Air  Test  Center 
Patuxent  River,  MD  20670 

1  Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  Snode 

Training  Analysis  &  Evaluation  Group 
(TAEG) 

Dept,  of  the  Navy 
Orlando,  FL  32813 
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Navy 


1  Dr.  Richard  Sorensen 

Navy  Personnel  R&D  Center 
San  Diego.  CA  92152 

1  Roger  Welsslnger-Baylon 

Department  of  Administrative  Sciences 
Naval  Postgraduate  School 
,  Monterey,  CA  93940 

1  Dr.  Robert  Wherry 
562  Mallard  Drive 
Chalfont,  PA  189 14 

,  1  Dr.  Robert  Wisher 
Code  309 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

1  Mr  John  H.  Wolfe 
Code  P310 

U.  S.  Navy  Personnel  Research  and 
Development  Center 
San  Diego,  CA  92152 
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Army 


1  Technical  Director 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  James  Baker 

Systems  Manning  Technical  Area 
Army  Research  Institute 
5001  Eisenhower  Ave. 

Alexandria,  VA  22 333 

1  Mr.  J.  Barber 

HQS,  Department  of  the  Army 
DAPE-ZBR 

Washington,  DC  20310 

1  Dr.  Beatrice  J.  Farr 

U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

1  Dr.  Michael  Kaplan 

U.S.  ARMY  RESEARCH  INSTITUTE 
5001  EISENHOWER  AVENUE 
ALEXANDRIA.  VA  22333 

1  Dr.  Harold  F.  O'Neil,  Jr. 

Attn:  PERI-OK 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  LTC  Michael  Plunmer 

Chief,  Leadership  &  Organizational 
Effectiveness  Division 
Office  of  the  Deputy  Chief  of  Staff 
for  Personnel 
Dept,  of  the  Army 
Pentagon,  Washington  DC  20301 

1  Dr.  Robert  Sasnor 

U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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Air  Force 


Marines 


1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Air  University  Library 
AUL/LSE  76/4*13 
Maxwell  AFB,  AL  36112 

1  Dr.  Alfred  R.  Fregly 
AFOSR/NL,  Bldg.  410] 

Bolling  AFB 
Washington,  DC  20332 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  AFB.  DC  20332 


1  H.  William  Greenup 

Education  Advisor  (E03D 
Education  Center,  MCDEC 
Quantico,  VA  221 3« 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  DR.  A.L.  SLAFKOSKY 

SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 


1  Dr.  Frank  Schufletowski 
U.S.  Air  Force 
ATC/XPTD 

Randolph  AFB,  TX  781 48 

2  3700  TCHTW/TTGH  Stop  32 
Sheppard  AFB,  TX  76311 


CoastGuard 


Other  DoD 


Chief,  Psychological  Reserch  Branch  12  Defense  Technical  Information  Center 

U.  S.  Coast  Guard  (G-P-1/2/TP42)  Cameron  Station,  Bldg  5 

Washington,  DC  20593  Alexandria,  VA  2231  *4 

Attn:  TC 

1  Military  Assistant  for  Training  and 
Personnel  Technology 

Office  of  the  Under  Secretary  of  Defense 
for  Research  &  Engineering 
Room  3D  129,  The  Pentagon 
Washington,  DC  20301 

1  DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 
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Civil  Govt  Civil  Govt 


1  Dr.  Paul  G.  Chapin 
Linguistics  Program 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Susan  Chipraan 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr .  John  Mays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  William  J.  McLaurin 
66610  Howie  Court 
Camp  Springs,  MD  20031 

1  Dr .  Arthur  Helmed 

National  Intitute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20208 

1  Dr.  Andrew  R.  Molnar 
Science  Education  Dev. 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Joseph  Psotka 

National  Institute  of  Education 
1200  19th  St.  NW 
Washing ton, DC  20208 

1  Dr.  H.  Wallace  Sinaiko 
Program  Director 
Manpower  Research  and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  2231 4 

1  Dr.  Frank  Withrow 

U.  S.  Office  of  Education 
H00  Maryland  Ave.  SW 
Washington,  DC  20202 


1  Dr.  Joseph  L.  Young,  Director 
Memory  &  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 
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Non  Govt 


1  Dr.  John  R.  Anderson 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Anderson,  Thomas  H.,  Ph.D. 

Center  for  the  Study  of  Reading 
174  Children's  Research  Center 
51  Gerty  Drive 
Champiagn,  IL  61820 

1  Dr.  John  Annett 

Department  of  Psychology 
University  of  Warwick 
Coventry  CV4  7AL 
ENGLAND 

1  DR.  MICHAEL  ATWOOD 

SCIENCE  APPLICATIONS  INSTITUTE 
40  DENVER  TECH.  CENTER  WEST 
7935  E.  PRENTICE  AVENUE 
ENGLEWOOD.  CO  80110 

1  1  psychological  research  unit 

Dept,  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600,  Australia 

1  Dr.  Alan  Baddeley 

Medical  Research  Council 

Applied  Psychology  Unit 
15  Chaucer  Road 
Cambridge  CB2  2EF 
ENGLAND 

1  Dr.  Patricia  Baggett 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  CDR  Robert  J.  Biersner 
Program  Manager 
Human  Performance 
Navy  Medical  R&D  Command 
Bethesda,  MD  20014 


Non  Govt  - 


1  Liaison  Scientists 

Office  of  Naval  Research, 

Branch  Office  ,  London 
Box  39  FP0  New  York  09510 

1  Dr.  Lyle  Bourne 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80309 

1  Dr.  John  S.  Brown 

XEROX  Palo  Alto  Research  Center 
3333  Coyote  Road 
Palo  Alto,  CA  9*1304 

1  Dr.  Bruce  Buchanan 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  DR.  C.  VICTOR  BUNDERSON 
WICAT  INC 

UNIVERSITY  PLAZA,  SUITE  10 
1160  SO.  STATE  ST. 

OREM,  UT  84057 

1  Dr.  Pat  Carpenter 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  William  Chase 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Michel ine  Chi 

Learning  R&D  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  William  Clancey 

Department  of  Computer  Science 
Stanford  University  . 

Stanford,  CA  94305 
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1  Dr.  Allan  M.  Collins 

Bolt  Beranek  &  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  Ma  02138 

1  Dr.  Lynn  A.  Cooper 
LRDC 

University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  15213 

1  Thomas  L.  Cr and ell 
35  Leslie  Avenue 
Conklin,  NY  13748 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street,  N.V. 

Washington,  DC  20036 

1  Dr.  Kenneth  B.  Cross 
Anacapa  Sciences,  Inc. 

P.0.  Drawer  Q 

Santa  Barbara,  CA  93102 

1  Dr.  Diane  Damos 

Arizona  State  University 
Tempe ,  AZ  85281 

1  Dr.  Bnmanuel  Donchin 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  LCOL  J.  C.  Eggenberger 

DIRECTORATE  OF  PERSONNEL  APPLIED  R 
NATIONAL  DEFENCE  HO 
101  COLONEL  BY  DRIVE 
OTTAWA,  CANADA  K1A  0K2 

1  ERIC  Facility-Acquisitions 
<1833  Rugby  Avenue 
Bethesda,  MD  20014 

1  Dr.  A.  J.  Eschenbrenner 
Dept.  E422.  Bldg.  81 
McDonnell  Douglas  Astronautics  Co. 
P.O.Box  516 
St .  Louis,  M0  63166 


1  Dr.  Ed  Felgenbaum 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94305 

1  Mr.  Wallace  Feurzeig 

Bolt  Beranek  A  Newman,  Inc. 

50  Moulton  St. 

Cambridge,  MA  02138 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organ. 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 

1  DR.  JOHN  D.  FOLLEY  JR. 

APPLIED  SCIENCES  ASSOCIATES  INC 
VALENCIA,  PA  16059 

1  Dr.  John  R.  Frederiksen 
Bolt  Beranek  &  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Alinda  Friedman 

Department  of  Psychology 
University  of  Alberta 
Edmonton,  Alberta 
CANADA  T6G  2E9 

1  Dr.  R.  Edward  Geiselman 
Department  of  Psychology 
University  of  California 
Los  Angeles,  CA  90024 

1  DR.  ROBERT  GLASER 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O’HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Marvin  D.  Glock 
217  Stone  Hall 
Cornell  University 
Ithaca,  NY  14853 
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Non  Govt  Non  Govt 


1  Dr.  Frank  E.  Gomer 

McDonnell  Douglas  Astronautics  Co 

P.  0.  Box  516 

St.  Louis.  HO  63166 


1  Dr.  Daniel  Gopher 

Industrial  &  Management  Engineering 
Technion-Israel  Institute  of  Technology 
Haifa 


ISRAEL 

1  DR.  JAMES  G.  GREENO 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O’HARA  STREET 
PITTSBURGH.  PA  15213 

1  Dr.  Harold  Hawkins 

Department  of  Psychology 
University  of  Oregon 
Eugene  OR  97403 

1  Dr.  Barbara  Hayes-Roth 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica.  CA  90406 

1  Dr.  Frederick  Hayes-Roth 
The  Rand  Corporation 
1700  Main  Street 
Santa  Monica,  CA  90406 

1  Dr.  James  R.  Hoffman 

Department  of  Psychology 
University  of  Delaware 
Newark,  DE  19711 

1  Dr.  Kristina  Hooper 
Q ark  Kerr  Hall 
(diversity  of  California 
Santa  Cruz.  CA  95060 


1  Glenda  Greenwald,  Ed. 

"toman  Intelligence  Newsletter 
P.  0.  Box  1163 
Birmingham,  MI  48012 


1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle.  WA  98105 

1  Dr.  Ed  Hutchins 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

1  Dr.  Steven  W.  Keele 
Dept,  of  Psychology 
University  of  Oregon 
Eugene,  OR  97403 

1  Dr.  Walter  Kintsch 

Department  of  Psychology 
University  of  Colorado 
Boulder,  CO  80302 

1  Dr.  David  Kieras 

Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  85721 

1  Dr.  Robert  Kinkade 
Essex  Corporation 
3211  Jefferson  Street 
San  Diego,  CA  92110 

1  Dr.  Kenneth  A.  Klivington 
Program  Officer 
Alfred  P.  Sloan  Foundation 
630  Fifth  Avenue 
New  York.  NY  10111 

1  Dr.  Stephen  Kosslyn 
Harvard  University 
Department  of  Psychology 
33  Kirkland  Street 
Cambridge,  MA  02138 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 
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Non  Govt 


Non  Govt 


1  Dr.  Alan  Lesgold 
Learning  R&D  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Mr.  Merl  Male  horn 
Dept,  of  Navy 
Chief  of  Naval  Operations 
OP-113 

Washington,  DC  20350 

1  Dr.  Erik  McWilliams 

Science  Education  Dev.  and  Research 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Mark  Miller 

TI  Computer  Science  Lab 
C/'O  2824  Winter  pi  ace  Circle 
Plano.  TX  75075 

1  Dr.  Donald  A  Norman 

Dept,  of  Psychology  C-009 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Committee  on  Human  Factors 
JH  811 

2101  Constitution  Ave.  NW 
Washington,  DC  20418 

1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  Dr.  James  W.  Pellegrino 
University  of  California, 

Santa  Barbara 
Dept,  of  Psychology 
Santa  Barabara,  CA  93106 


1  MR.  LUIGI  PETRULL0 

2431  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  Dr.  Martha  Poison 

Department  of  Psychology 
Campus  Box  3^6 
University  of  Colorado 
Boulder ,  CO  80309 

1  DR.  PETER  P0LS0N 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDER,  CO  80309 

1  Dr .  Steven  E.  Poltrock 

Department  of  Psychology 

University  of  Denver 

Denver, CO  80208 

1  Dr.  Mike  Posner 

Department  of  Psychology 

University  of  Oregon 

Eugene  OR  97403 

1  DR.  DIANE  M.  RAMSEY-KLEE 

R-K  RESEARCH  &  SYSTEM  DESIGN 
3947  FIDGEM0NT  DRIVE 
MALIBU,  CA  90265 

1  Dr.  Fred  Reif 

SESAME 

c/o  Physics  Department 
University  of  California 
Berkely,  CA  9^720 

1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Mary  Riley 

LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 
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1  Dr.  Andrew  H.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill.  NJ  07974 

1  Dr.  David  Rumelhart 

Center  for  Human  Information  Processing 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  DR.  WALTER  SCHNEIDER 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  ILLINOIS 
CHAMPAIGN,  IL  61820 

1  Dr.  Alan  Schoenfeld 

Department  of  Mathematics 
Hamilton  College 
a inton.  NY  13323 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRR0 

300  N.  WASHINGTON  ST. 

ALEXANDRIA.  VA  22314 

*  1  Committee  on  Cognitive  Research 

f  Dr.  Lonnie  R.  Sherrod 
Social  Science  Research  Council 
605  Third  Avenue 
New  York.  NY  10016 

*  1  Robert  S.  Siegler 

Associate  Professor 
Carnegie-Mellon  University 
Department  of  Psychology 
Schenley  Park 
§  Pittsburgh,  PA  15213 


1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  DR.  ALBERT  STEVENS 

BOLT  BERANEK  &  NEWMAN,  INC. 

50  MOULTON  STREET 
CAMBRIDGE,  MA  02138 

1  Dr.  Thomas  G.  Sticht 

Director,  Basic  Skills  Division 
HUMRR0 

300  N.  Washington  Street 
Alex and r la. VA  22314 

1  David  E.  Stone,  Ph.D. 

Hazeltlne  Corporation 
7680  Old  Springhouse  Road 
McLean,  VA  22102 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Kikumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
Uhiversity  of  Illinois 
Urbana,  IL  61801 

1  DR.  PERRY  TH0RNDYKE 
THE  RAND  CORPORATION 
1700  MAIN  STREET 
SANTA  MONICA,  CA  90406 


1  Dr.  Edward  E.  Smith 

Bolt  Beranek  A  Neman,  Inc. 
50  Moulton  Street 
Cambridge,  MA  02138 
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1  Dr,  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55**55 

1  DR,  GERSHOM  WELTMAN 
PERCEPTRONICS  INC. 

6271  VARIEL  AVE. 

WOODLAND  HILLS.  CA  91367 

1  Dr.  Keith  T.  Wescourt 

Information  Sciences  Dept. 
The  Rand  Corporation 
1700  Main  St . 

Santa  Monica,  CA  90*»06 

1  DR.  SUSAN  E.  WHITELY 
PSYCHOLOGY  DEPARTMENT 
UNIVERSITY  OF  KANSAS 
LAWRENCE,  KANSAS  660H4 

1  Dr.  Christopher  Wickens 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  J.  Arthur  Woodward 
Department  of  Psychology 
University  of  California 


